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Abstract — In this paper we consider the lossy compression of 
a binary symmetric source. We present a scheme that provides 
a low complexity lossy compressor with near optimal empirical 
performance. The proposed scheme is based on 6-reduced ultra- 
sparse LDPC codes over GF(g). Encoding is performed by the 
Reinforced Belief Propagation algorithm, a variant of Belief 
Propagation. The computational complexity at the encoder is 
0(< d > .n. q. log 2 q), where < d > is the average degree 
of the check nodes. For our code ensemble, decoding can be 
performed iteratively following the inverse steps of the leaf 
removal algorithm. For a sparse parity-check matrix the number 
of needed operations is 0(n). 

I. Introduction 

In this paper we address lossy compression of a binary 
symmetric source. Given any realization y 6 {0, 1}™ of a 
Ber(i) source Y, the goal is to compress y by mapping it 
to a shorter binary vector such that an approximate recon- 
struction of y is possible within a given fidelity criterion. 
More precisely, suppose y is mapped to the binary vector 
x e {0, l} fe with k < n and y is the reconstructed source 
sequence. The quantity R = — is called the compression 
rate. The fidelity or distortion is measured by the Hamming 
distance d H (y,y) = ^Yh=i \Vi-Vi\- The goal is to minimize 
the average Hamming distortion D = E[d/f(Y, Y)] for any 
given rate. The asymptotic limit, known as the rate-distortion 
function, is given by R{D) = 1 - H(D) for any D E [0, 0.5] 
where H(D) = -D log 2 D-(l-D) log 2 (l-£>) is the binary 
entropy function. 

Our approach in this paper is based on Low-Density Parity- 
Check (LDPC) codes. Let C be a LDPC code with k x n 
generator matrix G and mxn parity check matrix H. Encod- 
ing in lossy compression can be implemented like decoding 
in error correction. Given a source sequence y, we look for 
a codeword y 6 C such that dff(y, y) is minimized. The 
compressed sequence x is obtained as the k information bits 
that satisfies y = G T x. 

Even though LDPC codes have been successfully used 
for various types of lossless data compression schemes [4], 
and also the existence of asymptotically capacity-achieving 
ensembles for binary symmetric sources has been proved [21], 
they have not been fully explored for lossy data compression. 



It is partially due to the long standing problem of finding 
a practical source-coding algorithm for LDPC codes, and 
partially because Low-Density Generator Matrix (LDGM) 
codes, as dual of LDPC codes, seemed to be more adapted 
for source coding and received more attention in the few past 
years. 

In [20], Martinian and Yedidia show that quantizing a 
ternary memoryless source coding with erasures is dual of the 
transmission problem over a binary erasure channel. They also 
prove that LDGM codes, as dual of LDPC codes, combined 
with a modified Belief Propagation (BP) algorithm can satu- 
rate the corresponding rate-distortion bound. Following their 
pioneering work, LDGM codes have been extensively studied 
for lossy compression by several researchers [8], [9], [15], 
[18], [19], [27]. In a series of parallel work, several researches 
have used techniques from statistical physics to provide non- 
rigorous analysis of LDGM codes [5], [12] and [24]. 

In terms of practical algorithms, lossy compression is still 
an active research topic. In particular, an asymptotically opti- 
mal low complexity compressor with near optimal empirical 
performance has not been found yet. Almost all suggested 
algorithms have been based on some kind of decimation of 
BP or SP which suffers a computational complexity of 0(n 2 ) 
[5], [9] and [27]. One exception is the algorithm proposed by 
Murayama [24]. When the generator matrix is ultra sparse, 
the algorithm was empirically shown to perform very near 
to the associated capacity needing 0(n) computations. A 
generalized form of this algorithm, called reinforced belief 
propagation (RBP) [2], was used in a dual setting, for ultra 
sparse LDPC codes over GF(2) for lossy compression [14]. 
The main drawback in both cases is the non-optimality of 
ultra sparse structures over GF(2) [8], [15], [24]. As we will 
see, this problem can be overcome by increasing the size of 
the finite field. 

Our simulation show that b-reduced ultra sparse LDPC 
codes over GF(q) achieve near capacity performance for 
q > 64. Moreover, we propose an efficient encoding/decoding 
scheme based on RBP algorithm. 

The rest of this paper is organized as follows. Section Ullre- 
views the code ensemble which we use for lossy compression. 



Section UTTl describes the RBP algorithm over GF(g). We also 
discuss briefly the complexity and implementation of the RBP 
algorithm. In section |IV] we describe iterative encoding and 
decoding for our ensemble and then present the corresponding 
simulation results in section [V] A brief discussion on further 
research is given in Section PVll 

II. LDPC CODES OVER GF(q) 

In this section we introduce the ultra sparse LDPC codes 
over GF(g). As we will see later, near capacity lossy com- 
pression is possible using these codes and BP-like iterative 
algorithms. 

A. (X,p) Ensemble of GF(q) LDPC codes 

We follow the methods and notations in [16] to construct 
irregular bipartite factor graphs. What distinguishes GF(q) 
LDPC codes from their binary counterparts is that each edge 
of the factor graph has a label hij 6 GF(q) \{0}. In other 
words, the non-zero elements of the parity-check matrix of a 
GF(q) LDPC codes are chosen from the non-zero elements of 
the field GF(q). Denoting the set of variable nodes adjacent 
to a check node j by Af(j), a word c with components in 
GF(q) is a codeword if at each check node j the equation 

E ieA A(i) h i,j°i = holds - 

A (A, p) GF{q) LDPC code can be constructed from a 
(A, p) LDPC code by random independent and identically 
distributed selection of the labels with uniform probability 
from GF(q)\{0} (for more details see [1]). 

B. Code Construction for Lossy Compression 

It is well known that the parity check matrix of a GF(q) 
LDPC code, optimized for binary input channels, is much 
sparser than the one of a binary LDPC code with same 
parameters [1], [6], In particular, when q > 2 6 , the best 
error rate results on binary input channels is obtained with 
the lowest possible variable node degrees, i.e., when almost 
all variable nodes have degree two. Such codes have been 
called ultra sparse or cyclic LDPC codes in the literature. In 
the rest of this paper we call a LDPC code ultra sparse (US) 
if all variable nodes have degree two and the parity check's 
degree distribution is concentrated for any given rate. It is 
straightforward to show that for a US-LDPC code defined as 
above check node degrees has at most two non-zero values and 
the maximum check node degree of the code is minimized. 

Given a linear code C and an integer b, a o-reduction of 
C is the code obtained by randomly eliminating b parity- 
check nodes of C. For reasons to be cleared in section |IV] 
we are mainly interested in o-reduction of GF(q) US-LDPC 
codes for small values of 6 (1 < b < 5). Note that by 
cutting out a parity check node from a code, the number of 
codewords is doubled. This increment of the codewords has an 
asymptotically negligible effect on the compression rate since 
it only increases by 1/n while the robustness may increase. 

GF(q) US-LDPC codes have been extensively studied for 
transmission over noisy channels [13], [7], [6]. The advantage 



of using such codes is twofold. On the one hand, by moving 
to sufficiently large fields, it is possible to improve the code. 

On the other hand, the extreme sparseness of the factor 
graph is well-suited for iterative message-passing decoding 
algorithms. Despite the state of the art performance of mod- 
erate length GF(g) US-LDPC channel codes, they have been 
less studied for lossy compression. The main reason being 
the lack of fast suboptimal algorithms. In the next section 
we present RBP algorithm over GF(q) and then show that 
practical encoding for lossy compression is possible by using 
RBP as the encoding algorithm for the ensemble of o-reduced 
US-LDPC codes. 

III. Reinforced Belief Propagation Algorithm in 
GF(<7) 

In this section first we briefly review the RBP equations over 
GF(q) and then we discuss in some details the complexity of 
the algorithm following Declercq and Fossorier [7]. 

A. BP and RBP Equations 

The GF(q) Belief Propagation (BP) algorithm is a straight- 
forward generalization of the binary case, where the messages 
are q-dimensional vectors. 

Let fi £ v f denotes the message vector form variable node v to 
check node / at the ith iteration. For each symbol a EGF(q), 
the ath component of fi^f is the probability that variable v 
takes the value a and is denoted by fj^Ja). Similarly, /jA v 
denotes the message vector from check node / to variable 
node v at the iteration I and fjA v (a) is its ath component. 
Also let Af(v) {A4 (/)) denote the set of check (variable) nodes 
adjacent to v (/) in a given factor graph. 

Constants are initialized according to the prior informa- 
tion. The BP updating rules can be expressed as follows: 

Local Function to Variable: 

**/„(<o« s n *4'/0) (i) 

Conf (0i/) (a) v'£M(f)\{v} 

Variable to Local Function: 

/'6JV(o)\{/} 

where Conf( Ki n (a) is the set of all configurations of vari- 
ables in A4(f) which satisfy the check node / when the value 
of variable v is fixed to a. We define the marginal function of 
variable v at iteration I + 1 as 

e^\a)<x^ v (a) J] /4». (3) 
SeM(v) 

The algorithm converges after t iterations if and only if for 
all variables v and all function nodes / 

up to some precision e. A predefined maximum number of 
iterations £ max and the precision parameter e are the input to 
the algorithm. 



RBP is a generalization of BP in which the messages from 
variable nodes to check nodes are modified as follows 

M^Ho) oc (g*(a)) 7( V(a) [] ^, u (o), 

f'£Af(v)\{f} 

where is the marginal function of variable v at iteration £ 
and 7(£) : [0, 1] — ► [0, 1] is a non-decreasing function. It is 
convenient to define 7 to be 

7(0 = 1 - 7o7i> 

where 70,71 are in [0,1]. Note that when 71 = 1, RBP is 
the same as the algorithm presented in [24] for lossy data 
compression. In this case it is easy to show that the only fixed 
points of RBP are configurations that satisfy all the constraints. 

B. Efficient Implementation 

Ignoring the normalization factor in (O, to compute all 
variable to check-node messages at a variable node of degree 
d v we need 0(q.d v ) computations. A naive implementation 
of GF(<?) BP has computational complexity of 0(d 2 .q 2 ) oper- 
ations at each check node of degree df. This high complexity 
is mainly due to the sum in ([T), that can be interpreted as 
a discrete convolution of probability density functions. Effi- 
cient implementations of function to variable node messages 
based on Discrete Fourier Transform have been proposed by 
several authors, see for example [25], [17], [1], [7] and the 
references within. The procedure consists in using the iden- 
tity 0»'eM(/)\W M»7 = ? 1 (iL'eA-U/AM-^l/V/))' 
where the symbol denotes convolution. 

Assuming q = b p , the Fourier transform of each mes- 
sage [i v , j needs 0(q.p) computations and hence the total 
computational complexity at check node / can be reduced 
into 0(d 2 .q.p). This number can be further reduced to 
0(d f .q.p) by using the fact that H v ' eM(f)\{v} T = 
Uv'eMin^i^Vf) /F{fJ. vf ), or alternatively by using the 
summation strategy described in [3] which has the same 
complexity but is numerically more stable. Therefore, the total 
number of computations per iteration is 0(< d > .q.p.n) 
where < d > is the average degree. 

IV. Iterative Lossy Compression 

In the following three subsections we first describe a simple 
method for identifying information bits of a 6-reduced US- 
LDPC code and then present a near capacity scheme for 
iterative compression (encoding) and linear decompression 
(decoding). 

A. Identifying a Set of Information Bits 

For ^-reduced US-LDPC codes, one can use the leaf removal 
(LR) algorithm to find the information bits in a linear time. 
In the rest of this section we briefly review the LR algorithm 
and show that 1 -reduction (removal of a sole check node) of 
a US-LDPC code significantly changes the intrinsic structure 
of the factor graph of the original code. 

The main idea behind LR algorithm is that a variable on 
a leaf of a factor graph can be fixed in such a way that the 



check node to which it is connected is satisfied [22]. Given a 
factor graph, LR starts from a leaf and removes it as well as 
the check node it is connected to. LR continues this process 
until no leaf remains. The residual sub-graph is called the core. 
Note that the core is independent of the order in which leaves 
(and hence the corresponding check nodes) are removed from 
the factor graph. This implies that also the number of steps 
needed to find the core does not depend on the order on which 
leaves are chosen. 

While US-LDPC codes have a complete core, i.e. there is 
no leaf in their factor graph, the 6-reduction of these codes 
have empty core. Our simulations also indicate that even 1- 
reduction of a code largely improves the encoding under RBP 
algorithm (see section [V). How RBP exploits this property 
is the subject of ongoing research. It is straightforward to 
show that a code has empty core if and only if there exists 
a permutation of columns of the corresponding parity-check 
matrix H such that ^ for i = j and hij = for all 
i > j. 

As we have mentioned, LR algorithm can be also used to 
find a set of information bits of a given US-LDPC code. At 
any step t of LR algorithm, if the chosen leaf is the only leaf 
of the check node ft into which it is connected, then its value 
is determined uniquely as a function of non-leaf variables of 
check node f t . If the number of leaves dt is greater than 1, 
there are 2 dt ~ 1 configurations which satisfy the check node 
after fixing the values of non-leaf variables. At each step of 
LR we choose a subset of d t — 1 leaves. This set is denoted 
by Ft and we call it the free subset at t th step. Note that 
there are dt free subsets among which we choose only one 
at each step. It is straightforward to show that the union of 
all free subsets F = UtF^ R is a set of information bits for a 
given US-LDPC code. 

B. Iterative Encoding 

Suppose a code of rate R and a source sequence y is given. 
In order to find the codeword y that minimizes dff(y,y), 
we will employ the RBP algorithm with a strong prior 
£t*(a) — exp(—LdH(y v , a)) centered around y. The sequence 
of information bits of y is the compressed sequence and is 
denoted by x. In order to process the encoding in GF(q), we 
first need to map y into a sequence in GF(q). This can be 
simply done by grouping b bits together and use the binary 
representation of the symbols in GF(q). 

C. Linear Decoding 

Given the sequence of information bits x, the goal of the 
decoder is to find the corresponding codeword y. This can be 
done by calculating the G T x which in general needs 0(n 2 ) 
computations. One of the advantages of our scheme is that it 
allows for a low complexity iterative decoding. The decoding 
can be performed by iteratively fixing variables following the 
inverse steps of the LR algorithm; at each step t only one non- 
information bit is unknown and its value can be determined 
from the parity check f t . For a sparse parity-check matrix, the 
number of needed operations is 0(n). 



V. Simulation Results 

A. Approximating the Weight Enumeration Function by BP 

Given an initial vector y, and a probability distribution P(c) 
over all configurations, the P-average distance from y can be 
computed by 



y, 



(4) 



where P(ci) is the set of marginals of P. On the other hand, 
the entropy of the distribution P is defined by 



S(P) = -£P(c)logP(c) 



(5) 



Even though it is a hard problem to calculate analytically 
both marginals and S(P) of a given code, one may approxi- 
mate them using messages of the BP algorithm at a fixed point 
[26]. Assuming the normalized distance is asymptotically a 
self-averaging quantity for our ensemble, S(P) represents the 
logarithm of the number of codeword at distance Dp(y) + 
0(1) from y. By applying a prior distribution on codewords 
given by exp(— Ldn (c, y)) one is able to sample the sub- 
space of codewords at different distances from y. 

Fig. Q] demonstrates the WEF of random GF(q) US-LDPC 
codes for rates 0.3, 0.5, and 0.7 and field orders 2, 4, 16, 64 
and 256. The blocklength is normalized so that it corresponds 
to n = 12000 binary digits. 
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Fig. I . The approximate WEF of GF(q) US-LDPC codes as a function of q 
for a same blocklength in binary digits. 

Though BP is not exact over loopy graphs, we conjecture 
that the WEF calculated for US-LDPC codes is asymptotically 
exact. This hypothesis can be corroborated by comparing the 
plot in Fig. [T] with the simulation results we obtained by using 
RBP algorithm (Fig. ©. 

B. Performance 

In all our simulations the parameter 71 of RBP algorithm is 
fixed to one and therefore the function 7 is constant and does 
not depend on the iterations. We also fix the maximum number 
of iterations into £ max = 300. If RBP does not converge after 



300 iterations, we simply restart RBP with a new random 
scheduling. The maximum number of trials allowed in our 
simulations is T max = 5. The encoding performance depends 
on several parameters such as 70, L, the field order q, and 
the blocklength n. In the following we first fix n, q and L, 
in order to see how the performance changes as a function of 
7o- 

1) Performance as a Function of jq: Our main goal is to 
show that there is a trade off, controlled by 70, between three 
main aspects of the performance, namely: average distortion, 
average number of iterations and average number of trials. 
The simulations in this subsection are done for a 5-reduced 
GF(64) US-LDPC code with length n = 1600 and rate R = 
0.33. The factor graph is made by Progressive-Edge-Growth 
(PEG) construction [13]. The rate is chosen purposefully from 
a region where our scheme has the weakest performance. The 
distortion capacity for this rate is approximately 0.1754. 

In Fig. |2] we plot the performance as a function of 70. For 
70 = 0.92 we achieve a distortion of D — 0.1851 needing 
only 83 iterations in average and without any need to restart 
RBP for 50 samples. By increasing 70 to 0.96, one can achieve 
an average distortion of 0.1815 which is only 0.15 dB away 
from the capacity needing 270 iterations in average. 




Fig. 2. Performance as a function of 70 for a PEG graph with n=1600 
and R=0.33. The averages are taken over 50 samples. (a) Average distortion 
as a function of 70- For 70 > 0.96 the RBP does not converge within 300 
iterations. (b)The average number of iterations. (c)The average number of 
trials, (d) The average number of iterations needed for each trial. Note that 
even though average number of iterations show a steep increase as a function 
of 70, the average number of iterations needed per trial increases only linearly. 

2) Performance as a function of R and q: Fig. [3] shows the 
distortion obtained by randomly generated 5-reduced GF(q) 
US-LDPC codes for q = 2, q = 16 and q = 256. The block 
length is fixed to n = 12000 binary digits. For each given 
code, we choose 70 and L so that the average number of 
trials does not exceed 2 and the average number of iterations 
remains less than 300. Such values of 70 and L are found by 
simulations. Under these two conditions, we report distortion 



corresponding to best values of the two parameters averaged 
over 50 samples. 
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Fig. 3. The rate-distortion performance of GF(q) LDPC codes encoded with 
RBP algorithm for q = 2, 16 and 256. The blockiength is 12000 binary digits 
and each point is the average distortion over 50 samples. 



VI. Discussion and Further Research 

Our results indicate that the scheme proposed in this paper 
outperforms the existing methods for lossy compression by 
low-density structures in both performance and complexity. 
The main open problem is to understand and analyze the 
behavior of RBP over ^-reduced US-LDPC codes. 

As we have mentioned, 6-reduction of a US-LDPC code not 
only provides us with simple practical algorithms for finding 
information bits and decoding, but also largely improves the 
convergence of RBP. It is interesting to study the ultra sparse 
ensembles where a certain fraction of variable nodes of degree 
one is allowed. 
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